DCT-Former: Efficient Self-Attention with Discrete Cosine Transform
نویسندگان
چکیده
Since their introduction the Trasformer architectures emerged as dominating for both natural language processing and, more recently, computer vision applications. An intrinsic limitation of this family "fully-attentive" arises from computation dot-product attention, which grows in memory consumption and number operations $O(n^2)$ where $n$ stands input sequence length, thus limiting applications that require modeling very long sequences. Several approaches have been proposed so far literature to mitigate issue, with varying degrees success. Our idea takes inspiration world lossy data compression (such JPEG algorithm) derive an approximation attention module by leveraging properties Discrete Cosine Transform. extensive section experiments shows our method up less same performance, while also drastically reducing inference time. This makes it particularly suitable real-time contexts on embedded platforms. Moreover, we assume results research might serve a starting point broader deep neural models reduced footprint. The implementation will be made publicly available at https://github.com/cscribano/DCT-Former-Public
منابع مشابه
Image Steganography Using Discrete Cosine Transform (DCT) and Blowfish Algorithm
Steganography is one of the methods of secret communication that hides the existence of message so that a viewer cannot detect the transmission of message and hence cannot try to decrypt it. It is the process of embedding secret data in the cover image without significant changes to the cover image. A cryptography algorithm is used to convert the secret messages to an unreadable form before emb...
متن کاملThe Discrete Cosine Transform (DCT): Theory and Application
Transform coding constitutes an integral component of contemporary image/video processing applications. Transform coding relies on the premise that pixels in an image exhibit a certain level of correlation with their neighboring pixels. Similarly in a video transmission system, adjacent pixels in consecutive frames 2 show very high correlation. Consequently, these correlations can be exploited ...
متن کاملEnergy-Efficient Discrete Cosine Transform on FPGAs
The 2-D discrete cosine transform (DCT) is an integral part of video and image processing; it is used in both the JPEG and MPEG encoding standards. As streaming video is brought to mobile devices, it becomes important that it is possible to calculate the DCT in an energy-efficient manner. In this paper, we present a new algorithm and processing element (PE) architecture for computing the DCT wi...
متن کاملJPEG Encoder using Discrete Cosine Transform & Inverse Discrete Cosine Transform
In the past decade, the advancement in data communications was significant during explosive growth of the Internet, which led to the demand for using multimedia in portable devices. Video and Audio data streams require a huge amount of bandwidth to be transferred in an uncompressed form. The objective of this paper is to minimize the number of bits required to represent an image and also the ac...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
ژورنال
عنوان ژورنال: Journal of Scientific Computing
سال: 2023
ISSN: ['1573-7691', '0885-7474']
DOI: https://doi.org/10.1007/s10915-023-02125-5